If You Can't Be With 
the Data You Love: 
And the Risks of Loving 
the Data You're With 


Neeta Fogg Irwin Kirsch 
Paul Harrington Anita Sands 
Ishwar Khatiwada Larry Hanover 


Center for Labor Markets ETS 
and Policy, Drexel University 


THE ETS CENTER FOR RESEARCH ON HUMAN CAPITAL AND EDUCATION 


~ 


19990100 | 
067911101010770am 
1010111004078 


1010101010" 
601111101090910' 101" 


via 444’ 


Table of Contents 


Table of Contents 


PRTC GS gacuitet aecwasvaresieciecunieciebigetacieaeanianeneaenneeaee 1 
ACKNOWICAEINGIIUS sscciasectiesiacdciseccisnreneeediancatiers 4 
gia eye [W( do] y eeeeeeeeeemerenrer ricer tte ecenor aren sterner errnter onto y 5 
A Word about the Dataxcccviened etrrenninets P 
Divergent Measures: Educational Attainment vs. 
ESS el ctl SHIM os asics scscaptstay suet Sercenraveaceecuceete asta veyes 8 
Essential Skills Deficits of the Working-Age 
POU THIONA cans ssichepnnaprastsy canvas totes vacueseasesneenieteuataias 11 
PROXY MGASUI GS iis esveidareivenceadiisemntneuiieriae 11 
Measuring and Comparing Skills Deficits 
LISI PIAAC Daa sgesscegsaces tasers anc caesassetads Maxess 11 
The Interrelationship between PIAAC Literacy 
Scores and Educational Credentials .............. 13 
The Interrelationship between PIAAC Literacy 
SKINS Id LOW Wed GS oo cciscscascsascasssnssnanaiermeress 18 
What is the Utility of the Essential Skills Proxy 
MBAS UNES oc iccssegnscsceleancaieiiciusiiesaceaddinterecunnatentebcaucenies 24 
[MMIC ENON: sesza bs cascasceanteatsecenceecthernetystaxtdercetesteceaueeeet 27 
F219) 81118 | b Geren ee Omer tenner astern caren eer reer et 28 
AD OUT TMG AUUMIOIS 's.cascs.coscersdstencassoegadieiaesaasesceasones 29 


This report was written by: 


Neeta Fogg 

Paul Harrington 
Ishwar Khatiwada 
Irwin Kirsch 

Anita Sands 

Larry Hanover 


The views expressed in this 
report are those of the authors 
and do not necessarily reflect 
the views of the officers and 
trustees of Educational Testing 
Service. 


Copyright © 2019 by 
Educational Testing Service. All 
rights reserved. ETS, the ETS 
logo, and MEASURING THE 
POWER OF LEARNING are 
registered trademarks of 
Educational Testing Service 
(ETS). All other trademarks are 
the property of their respective 
owners. 


September 2019 


ETS Center for Research on 
Human Capital and 
Education 


Research and Development 
Educational Testing Service 
Rosedale Road 

Princeton, NJ 08541-0001 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


1 Preface 


Preface 


Policymakers and others in the United States, driven by a reasonable belief that when 
individuals have more education they will be better prepared for today's economy and have a 
better chance at favorable life outcomes, have recently redoubled efforts to improve rates of 
attaining credentials. On the surface, these efforts appear to be paying off. Today, a greater 
proportion of high school students are graduating on time than ever before, and among 
those who do not, most end up with a high school degree by their early 20s. What's more, the 
data show that the millennial generation has obtained the highest levels of postsecondary 
education of any previous generation in U.S. history. 


However, as Shakespeare once observed, "All that glitters is not gold." Mounting evidence, 
including this new paper, suggest that data on educational attainment and results from an 
array of skills assessments are telling divergent stories. In fact, recent research that examines 
the literacy and numeracy skills of America's millennial generation finds high levels of skills 
deficits. What's more, according to research on trends in fourth- and eighth-grade scores in 
the National Assessment of Educational Progress (NAEP; better known as the "Nation's 
Report Card"), the pattern of increasing scores and decreasing achievement gaps among 
race-ethnicity groups, evident in the first few years of the 2000s, has leveled off. 


Data on education in the United States has been formally collected since the late 19th 
century when Congress first passed legislation establishing a department to monitor the 
nation's educational progress. Of course, what constitutes being educated—and closely 
related, the way to measure progress toward that goal—has varied over time. From its 
beginnings until the middle of the 20th century, information on education inputs—such as 
numbers of teachers, schools, and students, as well as enrollment and attainment 
data—were used to understand the nation's educational progress. But, with critical increases 
in the need for highly skilled workers rising rapidly toward the second half of the 20th 
century, especially after America's Sputnik moment—when the Soviet Union overtook our 
nation in the space race and concerns began to mount that public education wasn't meeting 
demands—a move toward including measures of skill proficiency, or outputs, was added!" 
While perhaps an oversimplification, one could say skills data emphasized the quality of 
educational progress, while attainment and enrollment data emphasized the quantity. 


However, as we advance deeper into the 21st century, two troubling and paradoxical shifts 
appear to be emerging. First, at a time when higher levels of skills are essential for the long- 
term success of individuals—and the nation as a whole—we appear to be losing ground 
despite rising enrollment and attainment. Second, and in some ways more perplexing, we are 
seeing a move away from seeking to ensure that educational attainment comes with skills. 


In this paper, the authors highlight the critical disadvantages of focusing on educational 
attainment as the sole or primary marker of productive ability. They do this by comparing 
two estimates of employed adults in the United States believed to have foundational skill 
deficits—one relies on educational attainment data as a proxy for skill deficiencies, while the 
other utilizes skills data from a large-scale assessment of adults. The underlying theory 
behind using educational attainment as a proxy for skills is, quite reasonably, that the more 
education you have, the less likely, on average, you are to have a skills deficit. With this 
estimate, for example, those who dropped out of high school would be classified as deficient, 
while those with a bachelor's degree would be assumed proficient. The second estimate, on 
the other hand, is based on actual skills information from the Programme for the 
International Assessment of Adult Competencies (PIAAC), an international large-scale 
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assessment of adult skills in literacy, numeracy, and problem solving in technology-rich 
environments. For this approach, lack of skills is determined by performance on a set of 
literacy and numeracy tasks, regardless of education level attained. 


Through their analysis of attainment and skills data, the authors reveal that using educational 
attainment as a proxy for skill deficits results in large errors in terms of both the size and 
composition of those with low skills. These findings suggest extreme caution when using 
educational attainment as the only or primary measure of the productive ability of American 
students and adults. 


Indicators such as educational attainment are created, adopted, and used because they are 
believed to measure what we care about. What's more, they provide a framework that guides 
our thinking, policies, and actions toward achieving important national, state, and local goals. 
The waning ability of educational attainment to accurately measure progress on what we 
care about—the population having essential knowledge and skills, as shown in this and other 
research—has vast consequences felt by those without the skills they need despite a degree 
that signals otherwise. It is lived by those who take initiative and pursue higher education but 
either aren't equipped to succeed or struggle in programs that do little to foster skill 
development, although they do acquire debt. It also raises critical questions about the 
desirability of public policies that focus on educational credentials without acknowledging the 
levels of skills that are associated with those credentials. 


Allowing students to attain degrees and diplomas with low levels of literacy skills, particularly 
as those skills become even more critical for successful navigation of our complex world, 
diminishes the fundamental promise of education. 


—Paul Harrington and Irwin Kirsch 
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Notes 


' NAEP is a measure of the academic proficiencies of 4th-, 8th-, and 12th-grade students. 
Surveys of math, reading, and writing of 4th and 8th graders are conducted every two years: 
See About NAEP: A Common Measure of Student Achievement, National Center for 
Educational Statistics, https://nces.ed.gov/nationsreportcard/about/. 
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Beginning in the 1980s, a series of reports were published that focused on the links 
between educational performance, literacy skills, and the U.S. economy including A Nation at 
Risk, (National Commission on Excellence in Education, 1983), Strategy for U.S. Industrial 
Competitiveness (The Committee for Economic Development, 1984); A Nation Prepared: 
Teachers for the 21st Century (Carnegie Corporation Forum on Education and the Economy, 
1986), and Toward a More Perfect Union (Ford Foundation Project on Social Welfare and the 
American Future, 1988). 
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Introduction 


Measures of educational attainment such as high school graduation and college enrollment 
rates are at center stage in discussions of, educational progress, frequently serving as 
benchmarks of high school performance.’ Almost across the board, those rates show 
encouraging signs. A greater proportion of high school students are graduating on time than 
ever before, while most who do not graduate on time end up with a high school degree by 
their early 20s, and an overwhelming majority of high school graduates are enrolling in 
college. Race-ethnic groups have shared in the upward trend, with high school completion 
rates rising the most for Black and Hispanic students, and increases in the college enrollment 
rate of Black, and especially Hispanic, students outpacing even those of White high school 
graduates.” 


If the goal is to increase and expand the overall level of human capital in the society, these 
measures paint a promising picture and bode well for Americans—and America's future. 
However, when the lens is focused more sharply on the actual skills held by students and 
adults in the United States, the image we see shifts dramatically. Many are leaving upper 
secondary and postsecondary education with degrees, yet they lack the skills they'll need for 
future success in education or the labor market. That means that although both policymakers 
and researchers tend to rely on educational attainment as a gauge for what our students 
know, the data are telling us a misleading tale. 


American employers place a high value on skills and effectively find and reward individuals 
with them. In particular, there are substantial independent earnings gains for those with a 
bachelor's or higher degree, and similarly large earnings gains for higher levels of literacy and 
numeracy proficiency, both for prime age full-time workers and for college graduates.* 


The other side of the coin is that obtaining a degree while lacking the skills it purports to 
represent is a significant problem. There is an earnings penalty to those who lack adequate 
levels of skills—in other words, skills matter. Furthermore, new findings from PIAAC reveal 
that poor literacy and numeracy skills contribute to adverse life outcomes beyond just purely 
earnings, including a greater risk of unemployment and not having health coverage. 


Over the past 20 years, we have witnessed significant gains in educational attainment, with 
little evidence of a commensurate gain in skills. For example, recent research that examined 
the literacy and numeracy skill proficiencies of the American millennial generation found that 
while it has achieved the highest level of educational attainment of any previous generation 
in American history, the share of this cohort with deficits in essential skills is troublingly 
large.” What's more, according to research on trends in fourth- and eighth-grade scores in 
NAEP,° the pattern of i increasing scores and decreasing achievement gaps among race- 
ethnicity groups, evident i in the first few years of the century, has leveled off—especially in 
the area of mathematics.’ 


An extreme example of this paradoxical relationship between attainment and skills has 
recently come to light in the City of Detroit. The School District was sued by students who 
argued that the district had operated in a way as to deprive them of their constitutional right 
to "access to literacy."® In 2017, NAEP reading and math scale scores of eighth-grade 
students in Detroit were lower than those of any other large city in the nation,’ with 96 
percent in math and 93 percent of Detroit's eighth graders in reading scoring at or below 
basic levels.'° Even though NAEP achievement levels state that scores at the basic level 
denote only partial mastery of the knowledge and skills required for proficiency i in reading 
and math, Detroit achieved a near 80 percent (78.2) on-time graduation rate in 2017. 
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The primary goal of this paper is to highlight the critical disadvantages of focusing on 
educational attainment data as a stand-in for the level and distribution of skills across the 
population, which has a real human cost when we fail to see that millions of Americans lack 
adequate skills despite their diplomas and degrees. 


First, we'll provide additional background on the nature of the problem of overreliance on 
educational attainment data as well as what those statistics tell us as far as graduation and 
college enrollment rates. Then we'll see what data from PIAAC, an international large-scale 
assessment of adult skills, independently shows. We will then proceed to demonstrate the 
problems intrinsic in using educational data as a proxy for skills. Then we will look through 
the lens of a methodology, derived from a reasonably well-crafted approach used in Detroit, 
that relies largely on educational attainment to see the story it tells about the size and 
composition of the American population lacking essential skills. Finally, we will use the 
current gold standard—measurements of literacy and numeracy from PIAAC—and show the 
wide difference between the two measures. * 


As we pursued this work, however, we were ever mindful of what the late, great Harvard 
economist James Medoff advised his graduate students: If you can't be with the data you 
love, then you should love the data you're with. The pragmatics behind loving the data you're 
with are clear from the wide number of studies carried out by economists and social science 
researchers who utilize proxies to represent an issue of interest. This makes some sense: 
Large-scale sample surveys of households that include direct measures of skills and that are 
statistically valid may be the data we love, but their development is quite costly and their 
availability quite scarce. Indeed, proxy measures and approximations—the data you're 
with—dominate social and economic research despite the mixed results. '° 
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A Word about the Data 


To bring a tailored meaning to this study, we evaluated the essential literacy standards skill 
levels for college graduates against a more stringent standard than we did in our analogous 
evaluation for those with no college degree. The reason for using different PIAAC 
performance standards for these two groups is that different levels of skills are needed for 
occupations tied to college-level skills than for occupations that are not. Although measures 
of the employment and earning advantages of earning a bachelor's degree abound, careful 
studies of these advantages find that gains accrue only to college graduates who find 
employment in college labor market occupations, generally in a professional, managerial, or 
technical field that utilizes their more advanced education. College graduates who fail to find 
employment in the college labor market have earnings that are about equal to their high 
school graduate counterparts. “ 


Please also note that we limited our analysis of the PIAAC data to employed persons, as we 
were interested in measuring the size of skill deficiencies within the nation's 
workforce—those who have passed a basic labor market test and found work. Adults who 
were unemployed or not in the labor force were excluded from our study. 
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Divergent Measures: Educational Attainment vs. 
Essential Skills 


A 2006 California appeals court decision regarding a state legislative requirement that 
students pass an exit exam to receive a high school diploma delineated well the differences 
between educational attainment and skills: 


66 


The purpose of education is not to endow students with 
diplomas, but to equip them with the substantive knowledge 
and skills they need to succeed in life. A high school diploma 
is not an education, any more than a birth certificate is a 
baby. Its purpose is to symbolize the holder's acquisition of a 
certain level of knowledge and skills. Students who 
successfully completed their high school educations, but who 
did not receive diplomas for some reason (for example, 
because their school records were destroyed in a natural 
disaster), would still in fact possess the same level of 
education as persons with high school diplomas. As we have 
observed earlier, on the other hand, students who did not 
successfully complete high school, but who were awarded 
their diplomas anyway, would not, in fact, have acquired a 
high school education. ' 


99 


Despite the remonstrations of the California appeals court, the use of educational attainment 
and other common sample survey measures as a proxy for human capital is widespread. 
Data on educational attainment, earnings, and poverty status, along with other indicators, are 
widely available at the national, state, and local level through a number of federal statistical 
programs, most commonly the U.S. Census' Current Population Survey (CPS) and its 
American Community Survey (ACS). Yet direct measures of skills of adults are not as readily 
available. Since the early 1990s, there have been only a handful of surveys of adult literacy in 
the United States, and they have been conducted only about once a decade or so. ' 


So, as much as researchers and analysts might love to rely on the skills results, those who 
want to use such solid data are constrained by the data and measures contained in the 
public-use data files provided by statistical agencies. Moreover, the sample sizes of both the 
CPS and ACS surveys are large compared to sample surveys measuring adult skills that have 
been conducted in the United States in the past. © In short, statistically valid measures of a 
wide range of demographic, economic, social, and housing traits of the resident population 
for states and localities are readily available from these surveys, but data on adult skills are 
hard to come by. 


Researchers interested in understanding human capital characteristics and their role in the 
labor market usually must rely on measures of educational attainment instead of direct 
measures of literacy and numeracy proficiencies. Reliance on proxy measures of skills are 
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especially necessary at the state and local levels as there are essentially no comparable 
measures of literacy and numeracy skills of the adult population available, including for 
population and labor force studies. 


Indeed, much of the research literature related to the economics of human capital 
investment relies on the level of educational attainment as a proxy for human capital traits, 
including literacy and numeracy. The use of educational attainment as a measure of 
workforce skills or state or regional talent pools is commonplace, and economic development 
organizations regularly focus on the educational attainment of the employed in a region. ' 


The authors of this paper have shared the experience of relying on proxy measures of skills 
of the working-age population and labor force when direct measures were not available. 
However, as we demonstrate in this paper, the use of proxy measures of skill can result in 
large misclassifications of individuals as being skills deficient or not. 


But first, let's obtain a fuller picture of what measures of educational attainment show. It was 
in 2010-2011 that the U.S. Department of Education created a standardized measure of "on 
time" high school completion for students entering ninth grade: the adjusted cohort 
graduation rate. Although the use of educational attainment as a proxy measure of skills has 
been standard practice dating back decades, when economists began to examine population 
characteristics to help explain differences in productivity and living standards around the 
world,”° this was the first time that careful and systematic measures of high school 
completion across states and localities became available for analysis.*' With their improved 
quality, the use of these measures exploded. 


What they show is that the proportion of high school freshmen who graduated high school 
on time (in four years) has increased considerably since national and state data were first 
collected. The rate increased from 79 percent in 2010-2011 to 84 percent by 2015-2016. 


While not all high school freshmen graduate on time, most eventually earn a high school 
diploma or its equivalent. The 2009 High School Longitudinal Study of students entering ninth 
grade in 2009 found that by 2016, 92 percent had earned a diploma or equivalent.** Findings 
from the CPS indicate that the share of 18- to 24-year-olds with a high school diploma or 
equivalent increased from 88 to 93 percent from 2005 to 2015. Especially large gains in high 
school attainment were found among Black (86% to 92%) and Hispanic (70% to 88%) young 
adults. 


As for college enrollment, the national state and local data on college enrollment available 
are less standardized. They include the proportion of seniors taking SAT® and ACT® tests 
and, in some instances, findings from National Student Clearinghouse (NSC®) cohort studies 
of graduating senior classes of high schools in states.** National data on immediate college 
enrollment of graduating high school seniors are measured in the October Supplement to 
the CPS. The CPS findings reveal that in recent years, the proportion of high school graduates 
who enroll in college in the fall after completing high school has been relatively steady, 
hovering in the 67 to 70 percent range. But there have been gains in college enrollment by 
students in households closer to the bottom of the nation's income distribution, markedly 
narrowing income gaps in college enrollment. Differences in initial college enrollment by 
race-ethnicity have also narrowed, with especially large gains in postsecondary enrollment 
among recent Hispanic high school graduates.”> 
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While about two-thirds of high school graduates enroll in college right after high school, a 
substantial share of graduates delays enrollment. The Educational Longitudinal Study of 2002 
found that about 85 percent of high school sophomores eventually enroll in college.*° Thus, 
it seems that achieving the goal of "college for all" is close at hand. 


As we mentioned, one would ordinarily expect a commensurate improvement in skills to 
accompany these recent gains in enrollment and attainment. But, when skills data from U.S. 
and international surveys of students and adults are examined, a troubling pattern emerges. 
Emma Brown, an investigative reporter for the Washington Post®, published an article in April 
2016 in which she noted that while 82 percent of high school seniors graduated on time in 
2014, NAEP data revealed that only 37 percent were judged to be academically prepared for 
college coursework in reading.*’ Similarly, the percentage of 12""-grade students who scored 
proficient or better on 12"-grade NAEP in math and science ranged between 20 and 30 
percent. A similar concern regarding overall skill levels among U.S. millennials (young adults 
ages 16-34) was recently reported. U.S. millennials ranked last in numeracy skills, along with 
Spain and Italy, on the recent PIAAC international assessment of adult skills, and 16" out of 
22 countries in literacy.” 
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Essential Skills Deficits of the Working-Age 
Population 


Proxy Measures 


It was a 2018 report from Detroit, which sought to enumerate the city's adult population with 
essential skill deficits,’ that gave us the means of comparison and the impetus to compare 
"the data we're with" (educational attainment) to "the data we love" (skills). The motivation of 
the authors for their report from the Corporation for a Skilled Workforce, a Michigan-based 
organization dedicated to increasing economic opportunity and prosperity, was similar to our 
motivation for this one. They wrote: "The consequences of ... foundational (essential) skills 
gaps are profound. For employers, this greatly limits the number of Detroiters who possess 
essential competencies required at hiring. For job seekers with skill gaps, it means their 
opportunities for getting a good job and having opportunities to advance are few. For the 
city, it means a large proportion of residents struggle to sustain themselves and their 
families, with many forced to rely on public assistance to survive." 


Since no direct measures of adults' skills were available for the population of Detroit (or any 
other state or locality in the nation), the study authors' only viable option was to develop local 
proxy measures based on variables included in the ACS public-use microdata sample records 
for the city.>" 


In an attempt to get at skill deficiencies, then, the authors used the following available, 
mutually exclusive measures as proxies for skill deficits. There were four categories. For our 
purposes, we show only two of them as they represent 95 percent of what the study found to 
be the skills-deficient population.” 


* Educational Attainment (alone): Adult residents who do not have a high school diploma or 
its equivalent. 

* Low Earnings: Residents with a high school diploma and no postsecondary schooling who 
have low wages. 


The findings of the Detroit skills study were grim. An estimated 200,000 working-age 
nonelderly city residents aged 16 to 64 were found to be skill deficient—more than 40 
percent of the overall resident population of Detroit in that age category. Looking in more 
detail at those 200,000 working-age, nonelderly residents—figures generated by using 
attainment (and attainment coupled with earnings) as a proxy for skills—more than half 

(51 percent) were categorized as such based on the "low earnings" category, their education 
status tied in with earnings. An additional 44 percent was based on "educational attainment 
alone." 


Measuring and Comparing Skills Deficits Using PIAAC Data 


However, does this educational attainment-based approach generate an accurate view of skill 
deficiencies in the adult population? PIAAC data provide us with a unique opportunity to 
address this question. Using PIAAC data would allow us to examine the effectiveness of using 
what seem to be reasonable and level-headed proxies to extrapolate the size and 
composition of the employed skill-deficient population in the United States. 
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The PIAAC study sample is large (and costly) with more than 9,000 respondents, and some 
3,700 more to be added to the data files from 2017. PIAAC data provides national estimates 
of literacy and numeracy skills for adults ages 16-65, along with key demographic variables 
including race/ethnicity, gender, labor-force status, and highest level of educational 
attainment, among a host of other variables. These data can be used to produce measures of 
skill proficiency of the adult population that can be connected to a vast array of personal 
characteristics, behaviors, and life outcome measures. For example, the extensive labor 
market information gathered in the PIAAC database, when linked to measures of skills, has 
provided important new insights into the impact of literacy and numeracy skills on the 
employment and earnings experiences of adults. Some of these studies have raised 
important questions about the desirability of public policies that focus on educational 
credentials with little consideration to skills development associated with those credentials.** 


The downside of PIAAC is that like earlier national skills surveys, its per-unit cost of response 
is quite high, and so it has a comparatively small sample size, which effectively rules out the 
development of direct, sample-based measures of adult literacy and numeracy at the state 
and local level.° 


For our analysis, we utilized the PIAAC sample for the United States to explore relationships 
between the skills proxy measures of educational attainment and earnings adopted by the 
Detroit study authors, and generated estimates of the size and composition of the employed 
population in the country with skills deficits for select education/earnings groups. 


We then compared these findings with actual measures of literacy skills for those same 
education/earnings groups. In other words, we compared the size and characteristics of the 
skills-deficient population based on proxy measures—the data we're with—with direct 
measures of actual literacy skill proficiencies of the adult population derived from PIAAC 
survey data—the data we love. 


We examined the actual literacy skills of U.S. workers in the two main categories of adult 
working-age population with skills deficits used in the Detroit study: a) those without a high 
school diploma or equivalent and b) those with a high school diploma but earning low 
wages. 


We contend that using proxy measures of skill deficits can introduce two kinds of 
misclassifications regarding both the size and composition of those with a skills deficit: 
erroneously including employed persons with higher skills in the skill-deficient workforce and 
erroneously excluding employed persons with low skills. Both types of misclassification alter 
not only the overall size of the skills-deficient population, but also the basic demographic 
composition of this population. This fact should be of concern to both researchers and 
policymakers in that the use of these types of proxies can lead to important policy errors 
including misallocation of financial resources for education, training, and related human 
capital investment purposes in the public interest. 
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The Interrelationship between PIAAC Literacy Scores and 
Educational Credentials 


Below, we examine literacy proficiency in the context of PIAAC. Literacy scores are grouped 
into five categories representing distinct levels of literacy proficiency: 


* Level 1 or below reflects the rather limited ability associated with being able to read a short 
text and locate a single piece of information or enter some personal information into a 
document. A score at this level signals skill in reading a short, simple paragraph, 
understanding basic vocabulary, and the meaning of sentences. For persons with no 
college degree, we include those who have literacy scores of level 1 or below in our 
measure of the number of employed persons with skill deficits. 

* Level 2 reflects the ability to read a more complex text, compare and contrast some 
information, and paraphrase and make low level inferences from a text. For persons who 
have earned an associate's, bachelor's, or advanced degree, we count all persons with 
literacy skills at or below level 2 as skills deficient since, as noted earlier, literacy skills 
above this level are needed to effectively participate in the market for jobs requiring 
college-level skills. 

* Level 3 signals the ability to read more lengthy and dense texts. Individuals can identify, 
interpret, and evaluate pieces of information requiring more complex levels of inference. 
Those proficient at this level can identify and disregard irrelevant text to find the correct 
answer. Level 3 proficiencies and higher have been found to be associated with more 
favorable economic and social outcomes.” 

* Level 4 is an indicator of the ability to integrate, interpret, and synthesize information from 
complex or lengthy documents. Complex inference and use of background knowledge is 
important to interpret and evaluate evidence and persuasive arguments. 

* Level 5 demonstrates an ability to seek and integrate information from a variety of dense 
text sources, synthesize this information and identify contrasting points of view, and 
evaluate evidence-based arguments. 


Using the PIAAC national data, we present findings about the overall distribution of employed 
U.S. adults across the literacy levels (we combine levels 4 and 5 due to sample-size 
restrictions among employed adults in the nation; Figure 1). It is important to note that the 
skill levels of employed persons are, on average, higher than those of the population of all 
adults. Labor-force participation itself is associated with higher levels of literacy skills relative 
to the general population. Also, among those adults who participate in the labor force, 
unemployed persons have lower literacy skills than those who are employed.*° 


Large differences in literacy skills among employed Americans were found in our analysis of 
the PIAAC data. Among employed adults, the PIAAC survey found that 14 percent 
demonstrated literacy proficiencies that placed them at or below literacy level 1 (Figure 1). 
Proficiency at this level reflects minimal abilities to access and identify information in text, to 
integrate and interpret this information, and to evaluate the information a reader gleans 
from a text or document, skills that are associated with life outcomes. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


Essential Skills Deficits of the Working-Age Population 14 


Figure 1: Distribution of Employed Adults by Literacy Score Level (Standard Error in 
Parentheses) 
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Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


Comparing using level of educational attainment alone: The first major measure of 
deficient skills used in the Detroit study is level of educational attainment. Findings in Figure 2 
present the connection between educational attainment and low skills by examining the 
educational attainment of workers with literacy skills at or below level 1. We find that among 
the 18 million working American adults with literacy skills at or below level 1, only one in 
three (33.3%) did not obtain a high school degree or equivalent. Two-thirds (66.7%) of all 
employed persons with the lowest literacy proficiency have a high school diploma or higher 
level of education. 
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Figure 2: Distribution of Employed Persons with Literacy Score at or Below Level 1, by 
Level of Educational Attainment (Standard Error in Parentheses) 
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Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


High school graduates with no postsecondary education make up almost 40 percent of 
employed adults with very low literacy scores. What is most surprising about these findings is 
the large share of the low literacy workers who have enrolled in the nation's postsecondary 
education system. One in four employed persons with very low literacy scores has enrolled in 
college at some time. Even more remarkably, one in ten received a college degree. 


As we noted previously, the nation has experienced a substantial rise in four-year high school 
completion rates in recent years without corresponding gains in NAEP 12-grade proficiency 
scores. This suggests, as do our findings, that academic credentials are no guarantee of 
literacy proficiency. 


A look at the findings in Figure 3 confirms the disconnect between credential attainment and 
skills. This figure examines the skill level of employed adults by their highest level of 
educational attainment and reveals a positive but varied connection between educational 
attainment and skills. Persons who complete college degree programs are generally expected 
to have considerably higher literacy skills. Nevertheless, PIAAC literacy skills data reveal that 
among employed persons with a bachelor's degree, about one in five (21.3%) have literacy 
skills scores below level 3. More surprisingly, one in seven (14.5%) of those with advanced 
degrees (master's or higher) score below level 3 on the PIAAC literacy scale. 
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Figure 3: Distribution of Employed Persons, Ages 16-65, by Highest Level of Educational 
Attainment and Literacy Proficiency (Standard Error in Parentheses) 
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Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


As described earlier, we've adopted two separate tiers of literacy standards for this study: a 
higher minimum literacy level for college graduates and a lower minimum literacy level for 
those with no college degree because the earnings premium for college graduates relative to 
a high school credential accrues only if they work in a field using college-level skills.°° 
Therefore, given the large qualitative differences in occupational skill requirements for 
college labor market occupations (as well as the accompanying employment and earnings 
advantages of employment in these occupations), we opted to utilize a minimum literacy of 
level 3 for college graduates based on requirements for employment in such occupations. We 
opted to use a minimum literacy of level 2 as the standard for those with less than a college 
education, based on the likelihood that these individuals will work in occupations not 
requiring the proficiencies demanded in the nation's college labor market. 


Still, it is useful to recognize that skills at or above level 3 are often adopted as a significant 
cut point in the data because of the association between skills at these levels and favorable 
economic and social outcomes.*° That two in three (65.9%) employed high school graduates 
with no college degree have literacy skills below this minimum (level 3) is a worrisome finding 
for those concerned with both income inequality and economic growth. 


When we examine the size of the employed adult population with low skills (Table 1), a 
disturbing picture materializes. More than 10 million employed adults have completed high 
school (without earning any type of college degree) but have literacy skills that place them 
below level 2, that is, they have very low skills. Less surprising, an additional 6.1 million 
employed high school dropouts fall into this category, for a total of 16.6 million below level 2. 
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Applying a higher literacy standard for those who earn a college degree, as discussed above, 
we find that 12.4 million college graduates demonstrate low literacy proficiency, achieving a 
score that places them below level 3.*' 


Adding the results from using these two standards, then, we estimate that nearly 29 million 
employed Americans do not meet minimal literacy skills levels that might reasonably be 
associated with their level of educational attainment. 


Table 1: Size of the Literacy Skills Deficient Employed Population in the U.S., by 
Highest Level of Educational Attainment, 2012—2014 


HIGHEST LEVEL OF EDUCATION NUMBER PERCENT STANDARD 


ERROR 


Level 1 or Below (No College Degree) 
Less than 12, or 12 (no HS 


9 
Diploma) 6,149,174 37.1% 2.1 
High School Graduate (No College ; 
Degree) 10,440,256 62.9% 2.1 
Total Skills Deficient (No College ‘ 
Degree) 16,589,430 100.0% Tt 

Level 2 or Below (College Grads Only) 

Associate's Degree 4,686,747 37.9% 2.9 
Bachelor's Degree 5,348,418 43.2% 3:3 
Master's or Higher Degree 2,346,937 19.0% 2.4 
Total Skills Deficient (College 12,382,101 100.0% t 
Degree) 
Total Employed but Skills Deficient 28,971,531 22.3% 1.0 
Total Employed/Not Skills ° 
Deficient 100,719,781 77.7% 1.0 
Total Employed 129,691,312 100.0% Tt 


T not applicable. 
Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


It's worth noting that while the literacy and numeracy scale scores are positively related, they 
are not perfectly correlated. If we were to adopt a somewhat broader measure of skill 
deficiency that included adults with low scores on either the literacy or numeracy scales, the 
number of workers with skill deficiencies actually would be considerably higher than the 
number based on the literacy measure alone. 


Clearly, using educational attainment as a proxy for skills paints a misleading picture when 
trying to estimate the size of the employed population with low literacy skills. The problem is 
compounded when we delve more deeply into the distribution of skills by, for example, racial 
and ethnic groups. 
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PIAAC Literacy Skills and Educational Credentials by Race-Ethnicity 


Understanding the differences between educational attainment and literacy skill proficiencies 
is especially important when the discussion is focused on communities with high poverty 
rates, where residents tend to be members of underserved—often minority—populations. 
Many of these communities lack access to the resources needed to create the large human 
capital gains required to eliminate disparities for their residents. 


As we've argued, relying on educational attainment to gauge skill deficits results in a sizable 
misreading of the skills attainment for the total employed population, but as we will 
demonstrate, these differences are far more pronounced for minority populations that are 
much more likely to live in resource-poor communities like Detroit, which has seen gains in 
on-time high school completion rates but little gain in measured academic proficiency scores. 
The findings in Table 2 compare the distribution of literacy skills for White/Asian workers with 
those of Black/Hispanic workers for each level of educational attainment below the college 
degree level. We did not examine White, Asian, Black, and Hispanic workers separately 
because of insufficient sample size across race-ethnic groups. Therefore, we decided to 
group Black and Hispanic, who are often labeled underserved minorities in social science 
research, and Whites and Asians, who are not considered underserved. 


The findings reveal large literacy gaps between employed White/Asian and Black/Hispanic 
adults within each level of educational attainment, gaps in important information that are 
masked if using educational attainment as a surrogate for adequate skills. 


High school dropouts: The PIAAC survey found that about 19 percent of Black/Hispanic adult 
workers in the United States were high school dropouts compared to 6 percent of White/ 
Asian employed adults. This represents a large difference in the share of adults without a 
high school diploma. However, when we compare the literacy levels of employed White/Asian 
high school dropouts with those of their Black/Hispanic counterparts, we find even more 
extraordinary differences. 
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Table 2: Comparison of Literacy Level Distribution within Non-College Degree 
Educational Attainment Levels between White/Asian and Black/Hispanic 
Employed Adults (Standard Errors in Parentheses) 


CATEGORIES OF ASAI A BLACK/HISPANIC PCT. POINT 
PROFICIENCY fo) Yd Od 4D) SHARE OF DIFFERENCE (COL. 1 


/EDUCATIONAL PERSONS (COL. 1) EMPLOYED PERSONS MINUS COL. 2) 
ATTAINMENT LEVEL (of) Ey) 


Less than 12 Years (Not a High School Graduate) 


Overall 6.3% (0.3) 18.6% (1.1) -12.3% (1.1)* 
Level 1 or below £(T) 74.6% (4.3) #(T) 
Level 2 49.3% (3.9) t(t) +(t) 
Level 3 25.0% (3.5) +(T) +(T) 
Level 4/5 ¥(t) #(T) #(t) 
High School Graduate 
Overall 23.2% (0.4) 29.5% (1.1) -6.3% (1.3)* 
Level 1 or below 16.0% (2.0) 35.9% (3.9) -19.9% (4.5)* 
Level 2 43.6% (2.4) 44.3% (4.4) -0.7% (5.1) 
Level 3/4/5 40.5% (3.0) 19.9% (3.7) 20.6% (4.7)* 


Note: Data for cells with "+" are suppressed as they did not meet the minimum ETS sample size of 62; "T" not 
applicable; "*" statistical significance at .01 level 
Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


At least three-quarters of employed White/Asian workers without a high school diploma or its 
equivalent had literacy proficiencies scores at level 2 or above. That's in stark contrast to the 
three quarters of Black/Hispanic workers whose scores were below level 2. 


High school graduates: A closer look at Table 2 reveals that the size of the literacy skills 
difference by race-ethnicity is very large not just among those who left high school without a 
degree or its equivalent, but also among high school graduates. Among employed high 
school graduates, we find that 36 percent of employed Black/Hispanic graduates have 
developed only minimal literacy skills that place them below level 2, more than double the 
share of White/Asian high school graduates (16%). 


College graduates: The findings in Table 3 compare the distribution of White/Asian and Black/ 
Hispanic college graduates within different levels of college degrees by their literacy 
proficiency levels. The comparison reveals large gaps in literacy proficiency by race-ethnicity 
at the associate's, bachelor's, and master's degree level. About 19 percent of White/Asian 
graduates with an associate's degree or higher had scaled literacy scores that placed them 
below level 3 compared to more than twice as many (42 percent) of their Black/Hispanic 
counterparts. That is, those Blacks and Hispanics that have an associate's degree or higher 
are more likely to have very low proficiencies than their White/Asian counterparts. 
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Table 3: Comparison of Literacy Level Distribution within College-Degree 
Educational Attainment Levels between White/Asian and Black/Hispanic 
Employed Adults (Standard Errors in Parentheses) 


EDUCATIONAL WHITE/ASIAN SHARE OF BLACK/HISPANIC SHARE PCT. POINT DIFFERENCE 
ATTAINMENT EMPLOYED PERSONS OF EMPLOYED PERSONS (ole) Bs TUR of) Bay) 


AND LITERACY (oto) ED) (fo) Ey) 
LEVEL 


Associate's Degree or Higher 


Overall 47.6% (0.8) 25.6% (1.0) 22.0% (1.1)* 

Below Level 3 18.7% (1.6) 41.9% (3.3) -23.2% (3.7)* 

Levels 3/4/5 81.3% (1.6) 58.1% (3.3) 23.2% (3.7)* 
Bachelor's Degree or Higher 

Overall 37.3% (0.5) 16.9% (0.8) 20.4% (0.9)* 

Below Level 3 15.3% (1.6) 36.1% (4.1) -20.8% (4.4)* 

Levels 3/4/5 84.7% (3.5) 63.9% (6.8) 20.8% (7.7)* 


* Statistical significance at .01 level. 
Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


The Interrelationship between PIAAC Literacy Skills and Low Wages 


The second key indicator that the Detroit study uses to estimate the size of the skill-deficient 
population in Detroit is to assign that status to employed persons with high school degrees, 
but no postsecondary schooling, who earn low wages. Essentially, the Detroit study assumed 
that all low-wage workers with a high school degree were skill deficient. We examine this 
assumption using PIAAC data to measure the literacy proficiencies of workers at various 
points along the earnings distribution. Since the Detroit study does not indicate how they 
defined a low-wage worker and supplies no measures of the wages or earnings of these 
workers, we have defined low-wage workers as persons in the lowest 20 percent of the 
national earnings distribution as measured by monthly pay.” 


We used PIAAC data to create a monthly earnings distribution for all employed persons in the 
nation by ranking employed persons in the order of their monthly earnings. We then created 
five approximately equal-sized groups of these employed persons (earnings quintiles) and 
then computed the mean monthly earnings of all workers within each grouping. 


The findings of this analysis are presented in Figure 4. Similar to earnings distribution studies 
based on other data, the PIAAC data find large earnings differences between those at the top 
of the nation's earnings distribution and those at the bottom. Employed persons within the 
top fifth of the nation's monthly earnings distribution had average monthly earnings of over 
$10,700 at the time of the PIAAC survey administration. Those with earnings in the middle of 
the earnings distribution (third quintile, income between 40" and 60" percentile of the 
earnings distribution) had mean monthly earnings of $2,900. Persons working in jobs that 
placed them in the bottom fifth of the earnings distribution earned an average of just $755 
per month at the time of the PIAAC survey (Figure 4). “* 
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We use the bottom quintile to define low-wage workers. At $755 per month, this would mean 
annualized earnings of just over $9,000 per year for the mean worker in the bottom of the 
earnings distribution. 


Figure 4: Mean Monthly Earnings of Employed Persons by Earnings Quintile (Standard 
Error in Parentheses) 
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Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


The findings in Table 4 below examine the distribution of literacy skill levels of workers in 
each quintile of the earnings distribution in the nation. As the table illustrates, each earnings 
quintile is composed of about 26 million employed persons. The table examines how 
employed persons in each income quintile are distributed across literacy levels. 


The literacy proficiencies of workers in each earnings quintile are quite different. For 
example, we find that 45 percent of persons employed in low-wage jobs (the bottom quintile) 
have strong literacy proficiencies (level 3 and above), equal to the standard we expect for 
college degree holders. (In contrast, when we examine the literacy proficiencies of the highest 
earners in the nation, we find that more than one in five, or 22.4 percent, have skill scores 
below level 3). By relying on low earnings as a key criterion to identify those with weak skills, 
the Detroit study proxy approach includes a large number of persons with strong literacy 
skills who work in low-wage jobs, ma ny. in part-time positions. It's important to note that part- 
time employment is largely voluntary.” Students, retirees, and persons with family 
responsibilities often opt for part-time work, so it should not be surprising to find that a 
substantial share of workers with low monthly wages have strong literacy proficiencies.*° 
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Table 4: Distribution of Employed Persons within Each Earnings Quintile by Level 
of Literacy Proficiencies (Standard Error in Parentheses) 


LEVEL 1 OR LEVEL 2 AY] 3 ic AY] 3 7) NUMBER OF 


EMPLOYED 


EARNINGS 


QUINTILES BELOW 


PERSONS 


First 18.9% (2.1) 36.2% (2.5) 35.8% (2.9) 9.1% (1.3) 25,897,452 
Second 23.8% (2.1) 37.5% (2.3) 30.5% (2.0) 8.2% (1.3) 26,015,579 
Third 16.3% (1.7) 33.4% (2.6) 38.8% (2.3) 11.4% (1.3) 26,075,744 
Fourth 36.4% (2.4) 47.0% (2.6) 16.6% (1.9) 26,081,213 
Fifth 22.4% (2.3) 46.0% (3.1) 31.5% (2.5) 25,915,736 


Note: May not add up to 100% due to rounding. 
Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


Table 5 presents findings from the PIAAC survey on the literacy proficiencies of high school 
graduates with no college degree by their position in the earnings distribution of all workers. 
A look at the data reveals that including all low-wage workers at this level of education, as the 
Detroit study approach does, substantially overstates the size of the skills-deficient 
population in the nation. 


Nationally, the PIAAC data finds 8.1 million high school graduates (with no college degree) 
working at low wages. However, just 21 percent, or 1.7 million, of these low-wage high school 
graduates achieve literacy scores below level 2—the standard we might expect for high 
school graduates with no college degree. 


A second major limitation of relying on measures of low-wage workers as a proxy definition 
of skill deficits among employed high school graduates is the large number of adults with 
very low skills who are not included in the Detroit approach's low-pay measure of skill 
deficiency. Table 5 indicates that the Detroit approach fails to include (using our assumptions 
to define low income as the bottom quintile of monthly earnings) 5.4 million low-skill high 
school graduates with monthly earning levels that place them in all quintiles (second through 
fifth) above the bottom in earning distribution. 
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Table 5: Distribution of Employed High School Graduates (No College Degree) 
within Each Earnings Quintile by Literacy Score Level (Standard Errors in 
Parentheses) 


EARNINGS BELOW LEVEL 2 LEVEL 2 LEVEL 3, 4, &5 TOTAL 
QUINTILES EMPLOYED HS 
GRADUATES (NO 
COLLEGE 
DEGREE) 
Number 
First 1,725,806 3,283,444 3,130,430 8,139,680 
Second 2,099,379 3,503,270 2,091,670 7,694,319 
Third, Fourth, 
ar 3332677 7,218,132 5,728,210 16,279,019 
Total 7,157,861 14,004,847 10,950,310 32,113,018 
Distribution 
First 21.2% (3.2) 40.3% (4.0) 38.5% (3.8) 100.0% 
Second 27.3 (4.2) 45.5% (4.4) 27.2% (3.7) 100.0% 
ee oll 20.5% (2.4) 44.3% (3.4) 35.2% (3.7) 100.0% 
Total 22.3% (1.8) 43.6% (2.1) 34.1% (2.4) 100.0% 


Source: 2012/2014 U.S. PIAAC Surveys, Restricted Use File (RUF), tabulations by authors. 


In other words, defining skill-deficient high school graduates on the basis of their earnings 
results in the omission of nearly 5.4 million persons who have a high school diploma and low 
skills but monthly earnings that place them above the bottom one-fifth of the nation's 
monthly earnings distribution. This excludes 79 percent of high school graduates with the 
lowest literacy scores in the nation. If we were to apply a higher standard of skills proficiency, 
that is, include persons with level 2 literacy skills as having skills deficits, the Detroit skills 
deficit measure would exclude an additional 10.7 million high school graduates with level 2 
scores but earnings above those in the bottom one-fifth of the earnings distribution. 


Although the data in Table 4 do suggest a positive connection between skills and earnings, 
the findings in Table 5 suggest that this relationship is highly complex and that monthly 
earnings of workers are determined by many factors, ranging from the obvious, like full-time 
versus part-time work status, to more subtle earnings determinants, ranging from 
compensating wage differentials to work experience in differing labor market segments. 
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What is the Utility of the Essential Skills Proxy 
Measure? 


The Detroit study analysts were tasked with creating an estimate of the size and 
characteristics of the working-age population with essential skills deficits in the absence of 
direct measures of literacy and numeracy measures. The only reliable household survey that 
contains relevant measures that could be used as proxies for essential skills is the ACS. Given 
the questions on the ACS survey, the researchers made some reasonable assumptions about 
who is most likely to possess poor skills in the city. First, the authors assumed that 
educational attainment and skills are positively associated with one another (which, of 
course, they are) and so opt to identify those with the lowest level of educational attainment 
as a major part of those who lack skills. Second, the authors assumed that skills and the 
earnings of workers are positively correlated (which they are). For low earners, the authors 
took the further precaution of limiting this group to only low-wage workers with a high school 
diploma. 


Both of these assumptions appear level-headed and, in the absence of any direct measures 
of literacy or numeracy skills, are probably as good a set of proxy measures as any. Yet when 
we test these measures against the literacy proficiencies of adults in the nation by the same 
proxy educational attainment and low earnings status, we find that these measures do not 
serve well as an effective proxy for skills. 


Our analysis of PIAAC data found that 29.0 million employed persons in the nation had low 
literacy proficiencies. As can be seen on the left side of Figure 5, out of that total of those 
identified employed persons with low skills, 16.6 million had not earned a college degree, 
accounting for over half (57.3%) of the skills-deficient workforce in the nation (with literacy 
skills at or below level 1). Applying proxy measures omits 8.7 million (52.5%) of the 16.5 
million in this low-skilled employed population without a college degree (labeled as "less than 
associate's degree"). On the right side of the figure, we see that utilizing PIAAC data on skills 
identified an additional 12.4 million employed adults with low literacy skills who have earned 
a college degree. This group is completely excluded from the Detroit study measure of skill- 
deficient workers. Overall, utilizing the Detroit skills proxy measures excludes about 21 
million employed persons nationwide who are identified as skill deficient based on their 
literacy skill scores. 
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Figure 5: Employed Persons with Low Literacy Skills: PIAAC vs. Proxy Measures 
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The educational attainment/earnings proxy measure of skill deficits among employed adults 
also results in the erroneous inclusion of a large number of persons with stronger literacy 
skills (Table 6). When we applied the proxy measures to national data, we found that the 
measures wrongly included 6.4 million employed high school dropouts ("less than high 
school degree") in the skill-deficient workforce. These dropouts had literacy scores that 
placed them at level 2 or higher. Similarly, we found that 6.4 million low-wage high school 
graduates ("less than associate's") that are included under the Detroit study measure also 
had literacy scale scores that placed them at level 2 or above. Overall 12.8 million high school 
dropouts and low-wage high school graduates are erroneously included as skill deficient 
under the Detroit study measure. 
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Table 6: Estimated Number of Employed Americans, Ages 16-65, with Skill 
Deficiencies: Proxy Measure vs. PIAAC Skills Measure (in millions) 


CATEGORY PROXY PIAAC DIFFERENCE DIFFERENCE 
MEASURE SKILLS BY BY 
(>) aXe) a MEASURE* OMISSION COMMISSION 
INDICATORS) 
Less than High School Degree 12.5 6.1 T 6.4 
Less than Associate's (High School 
Degree [including Some College-No 8.1 1.7 T 6.4 


Degree]), Low Wage Earners 
Less than Associate's (High School 


Degree [including Some College-No 0 8.7 8.7 T 
Degree]) 

College Degree (Associate's and 0 12.4 12.4 t 
above) 

Total 20.6 28.9 


Total Difference: 


21.1 + 12.8 = 33.9 


T not applicable 

* Inclusion if: (1) below an associate's degree (less than high school or high school degree or some college, but no 
degree) and scored at or below level 1 on PIAAC, (2) attained an associate's degree or more, but scored below 
level 3. 


Figure 5 and Table 6 reveal two essential problems with relying on educational attainment 
and earnings status as proxy measures of skill deficits in the nation's workforce. The first 
problem is a sin of omission. The skills proxy measures approach results in failing to identify 
a large majority of the skills-deficient employed population in the nation. Indeed, we find that 
reliance on this measure missed over 21 million employed persons in the nation with skill 
deficits. 


The second problem is a sin of commission. The educational attainment/earnings measure 
results in the erroneous inclusion of 12.8 million employed persons with skills above those 
we would include in a PIAAC-based direct measure of low literacy skills. 


The combined magnitude of these misclassifications is large. In fact, the total absolute size of 
the combined omission and commission misclassifications in estimating the population of 
employed persons with a skills deficit is nearly 34 million employed persons. This not only 
represents a radical mismeasure of the size of the skills-deficient population, but also creates 
a troubling distortion of the composition of this population. 
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Implications 


Research and evaluation practitioners, especially those at the state and local level, are often 
pressed to create measures of economic and social outcomes that are important for 
developing education, workforce, and even economic development policy. Educational 
attainment has become the primary measure, and frequently the sole measure, of the 
human capital traits of the population of an area. Most social and economic policy decisions 
touch in important ways on the human capital proficiencies of the population—as measured 
by the level of educational attainment of adults. Yet the evidence is clear that attainment and 
skill, although positively related, are far from the same thing. While there has been increased 
emphasis on attainment as a measure of outcome for the nation's secondary and 
postsecondary system, there is reason to be concerned that there has been little corollary 
gain in literacy and numeracy skills of young people or adults. 


Surely, it's true that more of both credentials and skills can pay off in the labor market. 
However, as we have demonstrated, the connection between the two seems more tenuous 
than warranted by use of attainment as a proxy measure of skills. 


Researcher Mark Dynarski, discussing the recent rise in on-time high school completion rates, 
observed: "Today's elementary and middle school students will move on to high school, and 
one day they may graduate. Did they accomplish a real milestone, or were they just moved 
along? They are owed an answer."* 


Our answer to Dynarski's question is that today more than 10 million employed high school 
graduates were moved along with literacy skills at the basic or below level, and over 12 
million employed college graduates completed their college education, but did not achieve 
commensurate levels of proficiency. 


In other words, the extensive use of educational attainment as a measure of productive 
ability indicates that educational attainment is increasingly ineffective at assessing outcomes. 
Measures—statistics—are created, adopted, and used to track what we care about. They help 
guide our thinking, our policy, and our actions toward achieving that something. The waning 
ability of the educational attainment measure to represent productive ability has vast 
consequences. Consequences are felt by those individuals without the skills they need, 
although granted a degree that signals otherwise. It is lived by those who take initiative and 
pursue higher education but either aren't equipped to succeed or struggle in programs that 
do little to foster skill development, although they nevertheless acquire debt. And, it raises 
critical questions about the desirability of public policies that focus on educational credentials 
with little consideration to skills development associated with those credentials.”® 


However, we are all too aware that, as much as we love the comprehensive skills data that 
comes from a survey like PIAAC, its availability is limited. Data on educational attainment 
abounds. Our advice to researchers and policymakers, then, is simple. Love the attainment 
data you're with, but never unconditionally, and always be mindful of the types of variances 
that this report shows are apt to be present. Learn to realize that increases in graduation 
rates may be masking what's actually going on in the classrooms. And whenever you can get 
your hands on the skills data you love, use it as much as possible. It will improve your 
chances of getting better human capital investment policy in place. 
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Appendix 


Table A-1: Earnings Distribution of All Employed Nonelderly Adults in Detroit and 
Michigan 


QUINTILES MEDIAN EARNINGS 
DIST. 
Detroit 
1 5,617 5,562 243,508,013 43,351 3.6% 
2 15,918 15,572 705,147,173 44,298 10.4% 
3 26,285 25,892 1,136,841,177 43,250 16.8% 
4 37,816 37,414 1,516,810,889 40,110 22.4% 
5 75,452 63,705 3,175,944,545 42,092 46.9% 
Total 31,808 25,728 6,778,251,796 213,102 100.0% 
Michigan 
1 6,848 6,895 6,033,087,104 880,949 2.9% 
2 20,510 20,583 17,735,304,161 864,712 8.6% 
3 34,701 34,579 30,234,357,679 871,287 14.6% 
4 54,209 52,582 47,228,345,562 871,224 22.8% 
5 121,829 94,681 105,977,868,151 869,891 51.1% 
Total 47,546 34,380 207,208,962,656 4,358,063 100.0% 


Source: 2015, 2016, and 2017 American Community Surveys (ACS), public use files, U.S. Census Bureau, 
tabulations by authors. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


29 About the Authors 


About the Authors 


Neeta Fogg is Research Professor at the Center for Labor 
Markets and Policy at Drexel University; Paul Harrington is 
Professor and Director of the Center; Ishwar Khatiwada is 
an Economist at the Center. 


These authors have worked together for many years 
exploring various measures of human capital and the ways in 
which these human capital traits are valued in the labor 
market relative to other traits. Their research in this area has 
focused on four key measures of human capital: behavioral 
character traits, social skills, occupational knowledge, and 
literacy and numeracy. They have employed a variety of 

Neeta Fogg research techniques such as statistical analysis of large-scale 
databases with human capital measures, longitudinal 
analysis of educational progress and outcomes at the state 
and local level, net impact evaluation methods, and 
interviews with hundreds of employers, educators, organized 
labor leaders, and youth groups. 


A sample of their research and evaluation projects in recent 
years includes studies of the impact of disability on a variety 
of education and employment outcomes. They have also 
studied the nature of youth disconnection in cities across the 
nation. The authors have engaged in net impact evaluation 
studies of program interventions by high schools and 
workforce development programs. They have produced a 
series of papers on the relationship between college 

Paul Harrington education and labor-market outcomes of those with college 
degrees at the undergraduate and graduate level. Related to 
this has been a set of papers examining the underutilization 
in the U.S. labor markets of foreign-born college graduates 
with degrees earned abroad. A key research focus of the 
authors in recent years has been assessing impacts of an 
aging population on the labor force and its implications on 
the delivery of long-term care to this rapidly aging 
population. 


The authors have published numerous research 
monographs, book chapters, and articles in these research 
areas. Some of their recent publications include Gray 
Warnings: Challenges in the Direct Care Workforce; Opportunity 
Rising: Increases in Human Capital Investment and Declines in 
Ishwar Khatiwada Disconnection among Teens and Young Adults in Los Angeles; The 
Human Capital Investment Gap: Understanding the Diminished 
Prospects of Disconnected Youth in Los Angeles; From Diplomas 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


About the Authors 


es 


Tye 
¥ 


4: 14 


Ca 


| 
g a Wi " 


Ye 


Irwin Kirsch 


30 


to Degrees: A Longitudinal Study of College Enrollment and 
Graduation Outcomes of High School Graduates from the School 
District of Philadelphia; The 2018 Summer Job Outlook for 
American Teens; The Collapse of the Labor Market for 16- to 
24-Year-Olds; and U.S. Teens Want Work. 


Prior to 2011, the authors worked with Andrew M. Sum at the 
Center for Labor Market Studies at Northeastern University 
in Boston. 


Irwin Kirsch is the Ralph Tyler Chair in Large Scale 
Assessment and Director of the Center for Global 
Assessment at Educational Testing Service in Princeton, New 
Jersey. In his role as director of the center, he oversees 
several teams of research scientists and project managers 
who are responsible for the management, development, and 
implementation of various large scale international 
assessments as well as the creation of policy reports focusing 
on human capital and education. 


Over the course of his career, Kirsch has worked in close 
collaboration with a number of state, national and 
international organizations including the World Bank, 
UNESCO, the International Association for the Evaluation of 
Educational Achievement (IEA), and the Organisation for 
Economic Co-operation and Development (OECD), where he 
currently serves as the International Survey Director for both 
PISA (Programme for International Student Assessment) and 
PIAAC (Programme for the International Assessment of Adult 
Competencies). 


In addition to his assessment work, Kirsch is a member of the 
ETS Research Management Team, serves as an ad hoc 
reviewer for several journals, and has authored or co- 
authored a series of monographs for ETS on education, 
health, and workforce readiness issues. His most recent 
policy report, titled Choosing Our Future: A Story of Opportunity 
in America, seeks to provide understanding and communicate 
about the dynamics that govern the distribution and 
transmission of opportunity and proposes a framework for 
policies to expand opportunity for all Americans. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


About the Authors 


Anita M. Sands works as a researcher and author in the 
ETS Center for Research on Human Capital & Education. Her 
published work covers economic opportunity, racial and 
economic segregation, concentrated poverty, research 
methodology, and program evaluations. Most recently, Sands 
has co-authored three ETS reports—America's Skills Challenge: 
Millennials and the Future (2015), Choosing Our Future: A Story 
of Opportunity in America (2016), and Too Big to Fail: Millennials 
on the Margins (2018). Prior to joining ETS in 2011, Sands 
taught in the Department of Sociology at Rider University for 
nearly two decades and owned a consulting firm where she 
TERY 2 directed numerous projects on racial and economic 
segregation, poverty, and land-use policy. Sands is ABD in 
Sociology from Temple University. www.linkedin.com/in/ani- 
tasands 


Anita Sands 


Larry Hanover is an editor and author for the ETS Center 
for Research on Human Capital & Education. He also 
provides support for ETS's Center for Global Assessment, 
which oversees the Programme for the International 
Assessment of Adult Competencies (PIAAC) and the 
Programme for International Student Assessment (PISA). 
Larry previously worked as a reporter and editor for The 
Times of Trenton (NJ) and as an adjunct journalism professor 
at Temple University, and is the coauthor of a Holocaust 
memoir Rebuilt from Broken Glass. 


Larry Hanover 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


Endnotes 32 


Endnotes 


1 For an interesting discussion on college preparation, see: Cynthia G. Brown, "All Students Need Common 
Foundational Skills," https://www.educationnext.org/students-need-common-foundational-skills/; and Robert 
Schwartz, "Multiple Pathways Can Better Serve Students," Education Next, Summer 2014, https://www.education- 
next.org/multiple-pathways-can-better-serve-students/. 


2 The Condition of Education, U.S. Department of Education, National Center for Education Statistics, 2018, 
https://nces.ed.gov/programs/coe/pdf/coe_cpb.pdf. 


3 Neeta Fogg, Paul Harrington, and Ishwar Khatiwada, "Educational Attainment, Foundational Skills and Worker 
Earnings," New England Journal of Higher Education, January 15, 2019. https://nebhe.org/journal/educational-attain- 
ment-foundational-skills-and-worker-earnings/; Neeta Fogg, Paul Harrington, and Ishwar Khatiwada, Skills and 
Earnings in the Full-Time Labor Market (Princeton, NJ: Educational Testing Service, 2018), https://www.ets.org/re- 
search/report/earnings-full-time-labor-market/; Neeta Fogg, Paul Harrington, and Ishwar Khatiwada, Skills and the 
Earnings of College Graduates (Princeton, NJ: Educational Testing Service, 2019), https://www.ets.org/research/re- 
port/earnings-college-graduates. 


4 Ibid. 


5 Madeline J. Goodman, Anita M. Sands, and Richard J. Coley, America's Skills Challenge (Princeton, NJ: Educational 
Testing Service), https://www.ets.org/s/research/29836/; and Madeline Goodman and Anita Sands, Too Big to Fail: 
Millennials on the Margins (Princeton, NJ: Educational Testing Service, 2018), https://www.ets.org/research/report/ 
opportunity-too-big-to-fail/millennial. 


6 NAEP is a measure of the academic proficiencies of fourth-, eighth-, and 12""-grade students. Surveys of math, 
reading, and writing of fourth and eighth graders are conducted every two years: See About NAEP: A Common 
Measure of Student Achievement, U.S. Department of Education, National Center for Educational Statistics, 
https://nces.ed.gov/nationsreportcard/about/. 


7 Susanna Loeb, "Low-Performing Students No Longer Making Gains on NAEP," Education Next, April 10, 2018, 
https://www.educationnext.org/low-performing-students-no-longer-making-gains-naep/. 


8 Lorelie Laird, "Right to Learn Advocates for Detroit Students Say There's a Constitutional Claim to Literacy," ABA 
Journal, March 2017, http://www.abajournal.com/magazine/article/constitutional_right_to_learn. A similar right-to- 
access-to-literacy lawsuit against the state government is proceeding in California. See: Christine Hauser, "Too 
Many Children in California Can't Read, Lawsuit Claims," New York Times, December 6, 2017, https://www.ny- 
times.com/2017/12/06/us/california-literacy-lawsuit.html. 


9 Fourth-grade reading and math NAEP scale scores also placed Detroit last in fundamental skill proficiency scores 
among all big city school districts in 2017. See The Nation's Report Card, Data Tools District Profiles https://www.na- 
tionsreportcard.gov/profiles/districtprofile?chort=1 &sub=RED&Sj=XQ&sfj=NP&St=MN&year=2017R3. 


10 Basic level math scores mean that students understand arithmetic operations on whole numbers, fractions, and 
percentages. Students in the eighth grade with basic level reading scores are able to locate information, identify 
statements of main ideas, and make simple inferences from texts. 


11 "Michigan's Statewide Graduation Rate Hits 80 Percent Graduation, Graduation Rate Increases 0.53 Percent, 
Dropout Rate Declines," Michigan Center for Educational Performance, February 21, 2018 press release, available 
at https://www.michigan.gov/som/0,4669,7-192-26847-460814--,00.html. 


12 For a discussion of the PIAAC program in the United States, see "What is PIAAC?", U.S. Department of Education, 
National Center for Education Statistics, https://nces.ed.gov/surveys/piaac/. 


13 Proxy measurement of social and economic activity is not limited to the realm of human capital. One important 
example is the measurement and interpretation of consumer spending and its meaning for judging changes in 
the standard of living over time. See: Arnold Klingman, "The Post Industrial State," National Affairs, Summer 2018, 
https://www.nationalaffairs.com/publications/detail/the-post-industrial-state. 


14 Our college labor market classification is based on qualitative differences in the proficiencies required to perform 


effectively within an occupation, based on analysis of the U.S. Department of Education's O*NET System. For a 
description, see Fogg et al., Earnings of College Graduates. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


33 Endnotes 


15 Other studies, as one might expect, reveal that jobless adults have lower literacy and numeracy scores compared 
to their employed counterparts. However, if we were to include jobless adults in the study, the effect would be to 
disproportionately increase the measure of low-skill, nonelderly adults in the nation. 


16 O'Connell v. Superior Court, 141 Cal. App. 4th 1452, 1478, 47 Cal. Rptr. 3d 147, 167 (Cal. App 1 Dist. 2006). 


17 The National Adult Literacy Survey in 1992 was followed by the International Adult Literacy Survey in 1994, the 
National Assessment of Adult Literacy (NAAL) in 2003, and most recently PIAAC in 2012. 


18 For example, the CPS has a sample of about 60,000 households per month and the ACS sample includes about 
3.5 million households over the course of a year. In contrast, the PIAAC sample consists of about 9,000 
households, the NAAL sample about 19,000. 


19 In fact, a review of some of the proposal materials submitted by East Coast cities in the recent Amazon HQ2 
competition reveals that most highlighted the educational attainment of the regional workforce as well as the 
number or share of employed persons with a college degree. These measures served as proxies in these 
proposals for skills, talent pools, and similar synonyms for the stock of human capital. 


20 See Frederick Harbison, Human Resources as the Wealth of Nations (London: Oxford University Press, 1973); and 
Theodore Schultz, Investing in People: The Economics of Population Quality (Berkeley, CA: University of California 
Press, 1980). 


21 "Public High School Graduation Rates," The Condition of Education, Letter from the Commissioner, U.S. Department 
of Education, undated, https://nces.ed.gov/programs/coe/indicator_coi.asp. 


22 Alexandria Walton Radford, Laura Burns Fritch, Katherine Leu, and Michael Duprey, High School Longitudinal Study 
of 2009 (HSLS) Second Follow-Up, National Center for Education Statistics, U.S. Department of Education, February 
2018, https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=201 8139. 


23 National Center for Education Statistics, Digest of Education Statistics, 2016, Table 219.65, High School Completion 
Rate of 18- to 24-Year-Olds Not Enrolled in High School (Status Completion Rate), by Sex and Race-Ethnicity: 1972 
Through 2015. 


24 For example, the Rhode Island Department of Education has used NSC to track its public high school graduating 
seniors into the postsecondary system for a decade. NSC provides unit record data along with summary reports 
of immediate college enrollment after high school along with extensive records on postsecondary retention and 
completion to participating state and local education agencies and private schools. 


25 "Immediate College Enrollment Rate," The Condition of Education, January 2018, https://nces.ed.gov/programs/ 
coe/indicator_cpa.asp. 


26 Erich Lauff, Steven J. Ingalls, and Elise Christopher, Educational Longitudinal Study of 2002 (ELS2002): A First Look At 
2002 High School Sophomores 10 Year Later, U.S. Department of Education, National Center for Educational 
Statistics, January 2014, https://nces.ed.gov/pubs2014/2014363.pdf. It is useful to note that just 3 percent of these 
2002 sophomores had not earned a high school diploma or its equivalent. 


27 Emma Brown, "U.S. High School Seniors Slip in Math and Show No Improvement in Reading," Washington Post, 
April 27, 2016, https://www.washingtonpost.com/local/education/us-high-school-seniors-slip-in-math-and-show- 
no-improvement-in-reading/2016/04/26/9b8f033e-0bc8-1 1 e6-8ab8-9ad050f76d7d_story.html?noredi- 
rect=on&utm_term=.09e47b638fd6. 


28 Goodman, Sands, and Coley, America's Skills Challenge. 


29 The Detroit Adult Foundational Skill Development: Challenges and Solutions defines "foundational skills" (what we 
label here as "essential skills") as"... the basic, entry-level skills that are common across most workforces and 
industries, upon which most occupational skills are built. They are prerequisites for success in most jobs and 
careers. They include reading, math and English language proficiency; work readiness and basic professional 
skills; digital literacy skills; and career management skills." Mary Freeman, Larry Good, and Vickie Choitz, Detroit 
Adult Foundational Development, Corporation for a Skilled Workforce, June 2018, 4, http://skilledwork.org/wp-con- 
tent/uploads/2018/07/Detroit-Adult-Found-Skill-Development-6_18_18-1.pdf. 


30 Ibid. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


Endnotes 34 


31 Even up-to-date nationwide data is sparse. National sample surveys that directly measure adult literacy and 
numeracy are conducted infrequently. The PIAAC survey was begun in the United States in 2012. Prior to that, 
there had been only three other large-scale sample surveys designed to measure essential skills among adults 
conducted in the United States, beginning in 1992. Prior to PIAAC, the most recent was NAAL (2003). See History of 
International Adult Literacy Assessments, National Center for Education Statistics, https://nces.ed.gov/surveys/piaac/ 
history.asp. 


32 The other two are (a) Limited English-Speaking Proficiencies (but not Low Earnings), and (b) Combined Limited 
English and Low Earnings. Each group represented about half of the remaining 5 percent of Detroit's skills- 
deficient population. Note that the ACS limits questions about skills to a self-assessment of English-speaking skills 
for persons whose first language is not English. 


33 Tongyun Li, Matthias von Davier, Gregory Hancock, and Irwin Kirsch, The Predictions of Labor Force Status: 
Implications From the International Adult Skills Assessment, Research Report No. RR-16-11 (Princeton, NJ: Educational 
Testing Service, 2016), https://doi.org/10.1002/ets2.12097. 


34 However, it is possible to prepare indirect measures of state and even substate proficiencies using 
methodologies such as small area estimation. 


35 As noted earlier, we limited our analysis of the PIAAC data to nonelderly employed persons. This restriction 
paints a somewhat brighter picture of skills among adults as we opted not to present measures for all adults 
regardless of labor-force status. Adults who are unemployed or not in the labor force have significantly lower 
literacy and numeracy scores compared to their employed counterparts. 


36 These two categories accounted for most of the foundational skills deficit measure estimated in the Detroit 
study. 


37 Goodman et al., America's Skills Challenge; Organisation for Economic Co-operation and Development, Time for the 
U.S. to Reskill?: What the Survey of Adult Skills Says (Paris: OECD Skills Studies, OECD Publishing 2013), 
http://dx.doi.org/10.1787/9789264204904-en; U.S. Department of Education, Office of Career, Technical, and 
Adult Education, Making Skills Everyone's Business: A Call to Transform Adult Learning in the United States 
(Washington, DC: Author, 2015). 


38 Andrew Sum, Literacy and the Labor Force, Office of Educational Research and Improvement, U.S. Department of 
Education, National Center for Educational Statistics, 1999; Bobby D. Rampey, Robert Finnegan, Madeline 
Goodman, Leyla Mohadjer, Tom Krenzke, Jacquie Hogan, Stephen Provasnik, and Holly Xie, Skills of U.S. 
Unemployed, Young, and Older Adults in Sharper Focus, U.S. Department of Education, National Center for 
Educational Statistics, 2016, https://nces.ed.gov/pubs2016/2016039.pdf. 


39 Fogg et al., Earnings of College Graduates. 
40 Sands and Goodman, Too Big to Fail; Goodman, Sands and Coley, America's Skills Challenge. 


41 There were 1.9 million employed persons with an earned college degree who had level 1 or lower literacy scores, 
representing about 10 percent of all employed persons at this level. 


42 See for example: Sean F. Reardon, Demetra Kalogrides, and Ken Shores, The Geography of Racial/Ethnic Test Score 
Gaps, CEPA Working Paper No. 16-10, Stanford Center for Education Policy Analysis, 2019, http://cepa.stan- 
ford.edu/wp16-10; Adam Gamoran, The Future of Educational Inequality in the United States: What Went Wrong, and 
How Can We Fix It?", William T. Grant Foundation, 2015, https://wtgrantfoundation.org/library/uploads/2015/11/ 
The-Future-of-Educational-Inequality-Adam-Gamoran.pdf; Irwin Kirsch, Henry Braun, Mary Louise Lennon, and 
Anita Sands, Choosing Our Future: A Story of Opportunity in America (Princeton, NJ: Educational Testing Service, 
2016), https://www.ets.org/research/report/opportunity/choosing-our-future. 


43 The Detroit study found that about 104,000 employed residents with only a high school diploma were employed 
at low wages and thus classified as skills deficient. Total employment of the entire 16-and-over population in 
Detroit averaged about 225,000. The implied high share of employed high school graduates working at low wages 
is unsurprising as median annual earnings of the resident employed were just $39,500; statewide annual 
earnings averaged just over $69,000 during 2015. See Data Profiles for Michigan and Detroit Michigan, American 
Community Survey, https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2012/. 


44 For those interested in Detroit's and Michigan's earnings distributions, see the Appendix. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


35 Endnotes 


45 Over the last 20 years, an average of just over 80 percent of all part-time employment in the nation was 
voluntary: authors' analysis of Current Population Survey Public Use Micro sample data files, various years. 


46 There are reasons to believe that some share of involuntary part-time workers may be higher-skilled workers in 
occupations with low skill requirements, especially those dislocated from full-time jobs and subsequently re- 
employed involuntarily in part-time positions. See Henry S. Farber, Job Loss in the Great Recession and Its Aftermath: 
U.S. Evidence from the Displaced Workers Survey, NBER Working Paper No. 21216 (Cambridge, MA: National Bureau 
of Economic Research, 2015), https://doi.org/10.3386/w21216. 


47 Mark Dynarski, "Is the High School Graduation Rate Really Going Up?" Education Next, May 7, 2018, 
https://www.educationnext.org/is-high-school-graduation-rate-really-going-up/. 


48 Li et al., Labor Force Status. 


If You Can't Be With the Data You Love: And the Risks of Loving the Data You're With 


Measuring the Power of Learning? 


www.ets.org 


